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ABSTRACT 
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implement integrated standards -based science and mathematics teaching and 
assessment plans in their classes that are effective in helping students 
learn. The program provided professional development for 80 teachers in 37 
teams of science and mathematics teachers in 12 rural and urban school 
districts to develop and implement integrated teaching and assessment plans 
that follow the National Science Education Standards, Principles and 
Standards for School Mathematics, Benchmarks for Science Literacy, and 
Standards of Learning for Virginia Public Schools. Though the research 
investigated teaching and assessment, this paper focuses on strategies to 
enhance the teachers' critical analysis skills for assessing student 
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STRATEGIES ENABLING TEACHERS TO 
CRITICALLY ANALYZE LEARNING AND TEACHING 

Donna R. Sterling, George Mason University 




This paper shares the findings from four years of Eisenhower funded research which identified conceptual 
obstacles and enabling strategies for interdisciplinary teams of grade 4-12 teachers to develop and implement 
integrated standards-based science and mathematics teaching and assessment plans in their classes that are effective 
in helping students learn. The program provided professional development for 80 teachers in 37 teams of science 
and mathematics teachers in 12 rural and urban school districts to develop and implement integrated teaching and 
assessment plans that follow the National Science Education Standards (National Research Council, 1996), 
Principles and Standards for School Mathematics (National Council of Teachers of Mathematics, 2000), 
Benchmarks for Science Literacy (American Association for the Advancement of Science, 1993), and Standards of 
Learning for Virginia Public Schools (Board of Education Commonwealth of Virginia, 1995). Though the research 
investigated teaching and assessment, this paper focuses on strategies to enhance the teachers’ critical analysis skills 
for assessing student learning. 

As teachers change the way they teach to meet new national and state standards, they need to also change 
the way they plan for teaching and assessment of student understanding. The purpose of this research was to 
identify conceptual factors that limit teachers’ ability to successfully develop and implement effective teaching and 
assessment plans for their students. Once limiting factors were identified, enabling strategies were developed. The 
main areas for teacher professional development during the summer and implementation year were standards-based 
integrated science and mathematics subject matter and pedagogy, planning for teaching and assessment, and 
critically analyzing student learning (Scantlebury, Boone, Kahle, & Fraser, 2001). Though the science theme varied 
from year to year, an underlying focus on data analysis and experimental design remained. This presentation will 
focus on the support scaffolding that enabled teachers to more effectively critique their students’ learning. 

This study has implications for K-12 teacher professional development as we seek to help individual 
teachers and teams of teachers plan for standards-based teaching and assessment. As obstacles are identified and 
enabling strategies developed, teachers will be better able to plan and teach in ways called for in the state and 
national standards for science and mathematics. 
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Theoretical Underpinnings 



The study grew out of the recognition of the increasing importance for universities and schools to work 
together to support the learning and teaching of science and mathematics. It also grew out of the need to help 
teachers develop a vision of the kind of teaching and assessment called for in the national standards and the need to 
implement this type of learning and assessment in their classes (Anderson & Helms, 2001 ; Kahle, Meece, & 
Scantlebury, 2000; Lynch, 1997; Sterling, 1997, 2000, 2001; Sterling, Olkin, Calinger, Howe, & Bell, 1999). 

Since few changes usually take place as a result of professional teacher development (Guskey, 1995), we 
built into the program characteristics of "best practices" and “best of the best” for exemplary teacher professional 
development programs including a thematic design, supportive infrastructure, and utilization of evaluation (Ruskus, 
Luczak, & SRI International, 1995). A systemic approach was used that aligned curriculum, instruction, and 
assessment with local, state, or national standards and recruited teams of teachers from the same school and school 
division with the support of that division (Scantlebury et al., 2001). Additionally we focused on collaboration and 
follow-up (Gallagher, 1996; Ruskus, et al., 1995). Research suggests that meaningful collaboration facilitates 
educational reform (Anderson & Helms, 2001; Fullan, 1991; Keys & Bryan, 2001) and collaborative work cultures 
enhance student learning (Crawford, Kelly, & Brown, 2000; Newmann & Wehlege, 1995). 

To enhance the daily professional development environment of the summer workshops, many aspects of 
collaboration were built into the program (Keys & Bryan, 2001; Sweeney, Bula, & Cornett, 2001; Van Driel, 
Beijaard, & Verloop, 2001). Social learning theory suggests the importance of observing and modeling behaviors, 
attitudes, and emotional reactions of others as part of self-efficacy (Bandura, 1977). Therefore staff members were 
carefully chosen and provided with their own professional development training so that they became a team 
immersed in the projects culture. Vygotsky’s (1986) social development theory suggests that social interaction 
plays a pivotal role in cognitive development with peer collaboration exceeding what can be learned alone. Team 
problem solving and planning were an integral part of the program. According to Bruner (1960, 1990), learning is 
an active process where the learner constructs new knowledge by discovering principles themselves under the 
guidance of an instructor. Therefore instruction encouraged active dialog to uncover the structure and organization 
of new information in order for the learner to go beyond the information given (van Zee, Iwasyk, Kurose, Simpson, 
& Wild, 2001). Experiential learning situations were established through classroom experiences where the learners 
became personally involved in self-initiated activities when they designed and conducted their own research 
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investigations (Rogers, 1969). Cross (1981) emphasizes the importance for adult learning to be self-directed and 
problem-centered where they have as much choice as possible. Teacher teams were given the flexibility to adapt all 
assignments and research to their own schools and working situations. The perception of self-efficacy enhances 
cognitive development (Bandura, 1993, 1997). 

Initially the study focused on the scaffolding teachers needed to plan and teach standard-based science and 
mathematics. During this time, we realized that until teachers focused more on assessing student understanding few 
gains were likely to be made (Brown & Shavelson, 1996; Champagne, Lovitts, & Calinger, 1990; Kyle, 1997). 
While focusing on assessment, we realized that many teachers needed to be more critical about their students’ 
learning and their teaching. Our study has now been extended to focus on enhancing the teacher’s ability to 
critically evaluate learning and teaching. 

The immediate impetus for focusing on critical analysis of learning and teaching by teachers occurred when 
a team of teachers, who were reporting on a hands-on lesson where learning was not likely to have taken place, 
justified the success of their lesson by claiming their students had fun. Though fun is a desirable outcome from 
learning, it is not a replacement for learning. This particular team of teacher did not seen to have the knowledge and 
skills to critically analyze success in the classroom. Though many teachers are naturally reflective and critical about 
student learning, many are not. It became our goal to help all teachers critically analyze learning and teaching for 
the purpose of continually enhancing learning. 

Design and Procedure 

Program Design 

Structurally the program was set up to include teams of teachers collaboratively studying over an extended 
time period (Kahle, Meece, & Scantlebury, 2000; U.S. Department of Education, 1999; Van Driel et al., 2001). The 
program had an initial concentration of study and planning time for teachers in the summer followed by six to nine 
months of implementation, analysis, and sharing*of findings during the academic year (Anderson & Helms, 2001). 

The program was designed to provide participating teachers with professional development necessary to 
enable them to develop and implement integrated, hands-on, inquiry-based science and mathematics teaching and 
assessment plans (Parke & Coble, 200 1). During the summer workshops the teacher teams focused on developing 
integrated teaching plans that included the basic elements of experimental design and data analysis (Cothron, Giese, 
& Rezba, 2000; Virginia Department of Education, 2001). During the academic year the teachers focused on 
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implementing their plans and assessing their students growing understanding with support from their team members, 
other teams, and the instructional leadership team (Anderson & Helms, 2001; Sweeney et al., 2001; Van Driel et al., 
2001 ). 

Leadership Team 

The first phase of the program was to develop a leadership team that co-planned and taught the summer 
workshops and follow-up sessions. The team consisted of university faculty from science, mathematics, and 
education and teacher leaders from the different participating school divisions who were specialists in science or 
mathematics. Leadership skills were developed through increased knowledge of integrated science and mathematics 
gained by working with an interdisciplinary team during the planning and piloting process, critical analysis of 
student-centered teaching strategies and assessment practices, development and implementation of workshop plans, 
peer teaching and mentoring, and reflection and evaluation on every aspect of the program. 

Teacher Teams 

For this project, the ideal team was 2-3 teachers from the same school teaching the same grade level who 
could plan together. When this was not possible, teachers were allowed to choose to work with teachers from 
different schools but all at the same grade level or with teachers from different grade levels at the same school. 
Though we were not assessing effectiveness of team configuration, all arrangements appeared to enhance teachers’ 
experiences. Teacher connectivity and camaraderie appeared to be more significant than team configuration. The 
teacher teams created teaching plans that incorporated multiple forms of diagnostic, formative, and summative 
assessment to monitor student learning in their classes (Bell & Cowie, 2001; Treagust, Jacobowitz, Gallagher, & 
Parker, 2001). The research task for teachers was to identify a content standard and prove to their peers through 
assessment of understanding that their students had mastered the standard. If the standard was not mastered, they 
were to identify their students’ misunderstandings or misconceptions and their plans for enhancing understanding. 
Research Methodology 

Using a constant comparative process (Glaser, 1978; McMillan & Schumacher, 1984), data collected 
through surveys, interviews, focus groups, observations, and analysis of artifacts identified obstacles the teacher 
teams needed to overcome in developing integrated, inquiry-based science and mathematics teaching and 
assessment plans. A leadership team of scientists, mathematicians, and educators conducted the on-going formative 
research. The team analyzed the data on a daily basis during the summer program. This staffing arrangement 
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provided triangulation among the staff observations and interviews where staff members independently identified 
problems that were in most cases observed by others. 

Through a continuous improvement model, support scaffolding was developed that enabled teachers to 
effectively conduct research on their students understanding (National Commission on Mathematics and Science 
Teaching for the 2 1 st Century, 2000). The support scaffolding provided teachers with a simple way to assist in 
interpreting the complexity of teaching and constructing plans and hence assisted in the change process (Anderson 
and Helms, 2001; Barnett & Hodson, 2001). 

Findings 

Scaffolding for Planning 

The scaffolding that enables teachers to develop a vision of the kind of inquiry-based teaching and 
assessment called for by the standards and to effectively plan to create this type of teaching in their classroom fell 
into two categories - conceptual organizers and guided planning. Conceptual organizers in the form of graphic 
organizers proved to be especially helpful and were created to guide planning for teaching, assessment, and critical 
analysis. 

Obstacles and Enabling Strategies for Teaching and Assessment 

It became apparent that when teachers were developing their own teaching plans that were not based 
around a core set of materials such as a textbook, they were left with an organizing structural void. To fill this void, 
an inquiry-based conceptual organizer, a type of advanced organizer, provided an organizing structure/scaffolding 
around which to base teaching plans (Sterling, 2000). The inquiry hierarchy provided structure to both subject 
matter and pedagogical strategies. The inquiry hierarchy is similar to backmapping used to develop Benchmarks for 
Science Literacy (AAAS, 1993) in that it shows the relationship of unit science concepts. It is also similar to a 
problem-based unit that has a question guiding the instruction. 

Likewise it was found that teachers also needed an organizational structure around which to develop then- 
assessment plans. The assessment timeline conceptually organized a process for teachers to identify and monitor 
student learning (Sterling, 2001). By developing a before, during, and after paradigm of diagnostic, formative, and 
summative assessment, the teachers were able to embed assessment in their teaching. The teachers found that when 
they embedded assessment routinely as part of their instruction, they became more effective at assessing then- 
students understanding of science and in turn informing their instruction (Treagust et al., 2001). 
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Beyond the conceptual organizers, guided planning provided additional structure for the teachers that 
enabled them to create teaching and assessment plans which they could conceptually defend to their peers. The 
guided planning was a sequential series of decisions made by the teachers about the teaching plan or assessment 
plan they were developing, followed by an analysis of their decisions made from different perspectives. 

Obstacles and Enabling Strategies for Student Understanding 

While most teachers easily focused on assessing student understanding, peripherally related issues such as 
fun and active student involvement sidetracked some. Both fun and active student involvement are desirable 
outcomes. However, they may not be directly related to developing student conceptual understanding. 

To help teachers critically evaluate learning, a critical analysis taxonomy was developed showing a 
hierarchy of levels for analysis and evaluation (Figure 1). The taxonomy provides a structure for the purpose of 
continuous improvement for the teachers to evaluate effective learning and teaching. The taxonomy progresses 
from analyzing the effective aspects, to identifying the weaknesses, to suggestions for improvement, to extensions or 
links to other information. Using the taxonomy provided teachers with a conceptual framework for the evaluation 
process that delved at successively deeper levels of evaluation. 
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Figure 1. Critical analysis taxonomy. 

The critical analysis taxonomy also guided the teachers analysis when they implemented their teaching and 
assessment plans. By having teachers share all the levels of the critical analysis hierarchy, they celebrated their 
successes and group problem solved areas that needed further development. They became a team of professionals 
working together. 

Critical Analysis Taxonomy 

The critical analysis taxonomy provided a mental model for teachers to analyze student understanding. The 
taxonomy, a graphic organizer, combined the elements of a hierarchy with the need for depth of multiple formats of 
analysis. The reason for the hierarchy aspect was because the multiple levels for evaluation are based on a 
continuous improvement model and the research of Bloom (1956) and others that classifies thinking in a six level 
cognitive hierarchy from low to high level and that shows that people understand at many different levels. The 
hierarchy is also based on motivation systems theory and the need for positive regard for success (Ford, 1992). 
Continuous improvement is the goal but acknowledging what is working well is important so that it continues to be 
repeated. The continuous improvement model has the focus on improvement and not change for change sake. 
Therefore, it is important to include what is working so that it is not changed but continued. 

The critical analysis taxonomy provides a basis for evaluating most things and can be used by teachers as 
well as students. It could apply to a lesson being taught by a teacher or to evaluating an essay or poster by students. 
For a continuous improvement model the hierarchy builds from compliments, to criticism, to suggestions for 
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improvement, to extensions. Therefore it could be viewed as a cycle or spiral with each round of analysis informing 
the teacher and student about the level of understanding or lack of understanding. This in turn would inform 
instruction and learning, and focus on improvement. 

All levels of the hierarchy can be subdivided into two groups, critical analysis/comments that are 
peripherally related or that focus on core elements of effectiveness for the work being evaluated. Comments that are 
central to the effectiveness of the work being evaluated are the target for each level. However, by including 
peripherally related comments, a focus can be placed on honing in on core elements, but also acknowledging 
peripherally related comments and analysis. 

Pros 

The base of the hierarchy is identifying positive or successful aspects of the work being evaluated. Most 
people find it easier to give compliments than to criticize. An example of the two levels within the pros category 
when evaluating a teaching activity are comments about peripheral issues such as students having fun as compared 
to comments about student learning. Though having fun is desirable, it is not usually the central purpose for an 
activity. 

Cons 

Identifying aspects that are not working or are only partially working is the next step and a prerequisite to 
improving. It is generally more difficult for people to be forthcoming when analyzing what is not working than 
what is working because of values associated with lack of success. Therefore we deepened the “cons” part of the 
analysis to a problem-solving/data analysis paradigm where problems are solved (Cothron, Giese, & Rezba, 2000; 
Gabel, 2002) (see Figure 2). Analysis of errors of understanding is a cyclic process that starts with identifying errors 
and looking for patterns among identified errors. Analyzing reasons for errors and clarifying as needed by gathering 
more data about what led to errors enables teachers or students to plan for remediation. After remediation, the cycle 
starts again with identifying evidence of understanding or errors in understanding to determine if remediation has 
been successful. 

i \ 
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Figure 2. Error analysis. 




Establishing a safe and supportive environment that focuses on continuous improvement and rewards 
honest reflection is crucial to encourage sharing especially of less than stellar performance by students or teachers. 
As part of the sharing process, teachers shared samples of student work. In most cases they shared three samples, 
one each from the top, middle, and bottom third of their class. Inquiring into your own practice and sharing about 
research findings and dilemmas is part of the new inquiry group paradigm for professional development (National 
Commission on Mathematics and Science Teaching for the 21 st Century, 2000). 

Improvements 

After identifying what is not working, suggesting possible solutions to problems or ways to make 
something more effective is the next level. 

Extensions 

Extensions are ways that the work being assessed can be connected or extended to make it more 
meaningful. It tends to be value neutral and thus brings the focus back to quality teaching and assessment at all 
levels. 



The taxonomy proved to be most helpful in stretching teachers to go beyond accepting the status quo to 
improving learning and teaching. The teachers also found that the taxonomy could be used with students to help 
them with evaluation of their own and other students’ work. 

Conclusion 

This study identified conceptual obstacles for standards-based teaching and assessment and developed 
support scaffolding that enabled teachers to understand and accommodate into their teaching style a student-centered 
approach to assessment. The scaffolding included an assessment timeline and critical analysis taxonomy that 
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conceptually organized the assessment process and a series of assessment analyses that focused on the effectiveness 
of learning and assessment strategies. 

By conducting research on their own students’ understanding, the teachers appear to critically analyze the 
teaching and learning process. The teachers found that when they embedded assessment routinely as part of their 
instruction, they became more effective at assessing their students understanding of science during the teaching 
process especially when they used multiple forms of assessment. 

As new ways of teaching and assessing learning challenge traditional methodology, teachers need time to 
work through the conceptual change process. As the teachers are introduced to new methodologies and develop a 
new understanding of effective science teaching, they require multiple experiences that challenge their 
understanding of learning. A simple conceptual paradigm and a series of experiences that assists the teachers in 
investigating overtime the new strategies at ever increasing depths helps teachers to progress through the change 
process. 

By using the critical analysis taxonomy and conducting research on their own students understanding, most 
teachers appear to be able to critically analyze the teaching and learning process for their students. Our research 
identified conceptual obstacles for creating and evaluating standards-based teaching and assessment and developed 
scaffolding that enabled teachers to understand and accommodate into their teaching style a student-centered 
approach to hands-on, inquiry-based teaching and assessment that led to assessing and extending student conceptual 
understanding. 
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